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We study a model of a polling system, that is, a collection of d 
queues with a single server that switches from queue to queue. The 
service time distribution and arrival rates change randomly every 
time a queue is emptied. This model is mapped to a mathematically 
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■ equivalent model of a random walk with random choice of transition 

' probabilities, a model which is of independent interest. All our results 

are obtained using methods from the constructive theory of Markov 
chains. We determine conditions for the existence of polynomial mo- 
ments of hitting times for the random walk. An unusual phenomenon 
. of thickness of the region of null recurrence for both the random walk 

and the queueing model is also proved. 

\o ' 

CNJ ■ 1. Introduction. The results presented in this paper, depending on the 

affinities of the reader, can be looked at from two different viewpoints: a 
model from queueing theory or a model from the theory of Markov chains. 
These two models are mathematically equivalent but have their own indi- 
! vidual interest. In this paper we start by describing the queueing model and 

stating the results for it. Subsequently, the proofs are obtained, after the 
queueing model has been bijectively mapped to a random walk model, by 
using methods from the constructive theory of Markov chains to determine 
the asymptotic behavior of this random walk. A phenomenon, exceptional in 
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the theory of stochastic processes, namely, a thick region of null recurrence, 
is also proved for our model. 

Our main objective is to investigate stability conditions for a polling model 
with parameter regeneration. We feel that parameter regeneration is quite 
a natural hypothesis for realistic queueing systems: it models changes in 
behavior (of customers or servers) due to factors that are external to the 
studied system. In our model there are d > 2 stations where queues of jobs 
can form and a single server which switches from station to station. The 
server only processes jobs at its current station. For clearer exposition we 
treat in this paper the case where only two stations are open, that is, re- 
ceiving jobs, at any time. We study the model where all stations are open 
in a separate paper [9]. The reason for splitting this work into two sepa- 
rate papers, depending on whether two or more stations receive jobs, is that 
in the former case stability conditions are expressible in terms of explicitly 
determined factors, while in the latter, they are expressible in terms of fac- 
tors whose existence is shown but whose values are inaccessible to direct 
computation. These factors are reminiscent of the Lyapunov exponents for 
products of random matrices whose values can be approximated only by 
stochastic simulation. Moreover, while the obtained results in the two pa- 
pers can be phrased in similar ways, the techniques used to obtain them 
differ at key points. 

When the server completes all the jobs at its current station k, say, it 
starts to move to the other open station i, station k closes and indepen- 
dently of the process history, station j ^ i opens with probability Py , where 
J2j^i P%j = 1 f° r each station i. The sequence of stations visited by the server 
thus forms a Markov chain {I r :r = l,2,...}on state space = {1,2, ... ,d} 
with transition matrix P which we will assume is irreducible but may be 
periodic (this is necessary for the case d = 2 and for cases where one station 
has priority over the others). We call the time period between the server's 
arrival at and departure from a station an epoch. The server takes some time 
with distribution Hij which is independent of the process history to switch 
between stations and we will assume such times have bounded first moments 
(for some results a stronger assumption will be needed). 

During each switching time and service epoch, jobs arrive at the open 
stations in independent Poisson streams with rates Ai at the server's station 
i and A2 at the other open station j. The jobs at station i have independent 
service times with distribution function G, mean where G is selected 
from the class of distributions V with uniformly bounded second moments. 
This service time distribution and the arrival rates (Ai,A2) are regenerated 
at the start of each switching time independently of the process history using 
a measure Uy which depends upon the ordered pair (i, j) of open stations. It 
will follow from our Condition E, given later in the paper, that there exist 
two positive constants tuq and Mq, with mo < Mq such that tuq < Aj < Mq 
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for i = 1, 2. Each job at queue i leaves the system after processing by the 
server and when the server completes all jobs at station i, including any that 
arrive since the server switched to % (exhaustive service), the epoch ends. 
Succeeding epochs are defined in the same way except that if the system 
ever has no jobs, then the server waits for some to arrive — we do not go 
into any details as we are interested in establishing stability criteria, not 
studying equilibrium behavior. Our main interest is in the queue lengths, 
but their behavior is controlled by the process of regenerated parameters, 
so we consider this in detail. 

Parameter regeneration. The parameters in each epoch have the form 
(i, j, G, Ai, A2) E <Sj x T x (mo, Mo) 2 = O e . We impose below conditions on 
service times and arrival rates that ensure that the overall rate of events 
for the random walk is uniformly bounded and all epochs are finite with 
probability 1. 

As stated above, we assume that the Markov chain {I r } supplying the 
sequence of open stations is irreducible and starts from a state i\ € S^. De- 
note the space of parameter sequences by fts = = {(*r> *r+i> G r , \ r ) : r = 
1,2,...}, the collection of all sequences of ordered pairs of open stations 
together with service time distribution and arrival rates. 

Probability is associated with events in Q$ as follows. At each switch- 
ing time r > 1, the station where the server was closes, the server starts 
switching to station I r = i and at the same time, with chance Py, j = 1, 
. . . , d, station I r+ \ = j ^ i opens. The arrival rates X r = (Ai(r), A2(r)) and 
the service time distribution G r at queue i (the random parameters) are 
chosen using measure Vy independently of the process history and are con- 
stant throughout the epoch. As stated above, the server takes a time with 
distribution Hij to complete its switch from i to j. 

We will use oj 6 Q$ to denote a sequence of regenerated parameters and 
u; r = (i r , i r +i, G r , X r ) to denote the parameter set in epoch r. Where it will 
not cause confusion we will drop the epoch number dependence from pa- 
rameter sets and write u> r = (i,j,G,\). 

Embedded Markov chain. We find it convenient to establish our stability 
results for the queueing process in a discrete time setting, so we work with a 
Markov chain embedded in the continuous time process. Specifically, we con- 
sider the discrete time embedded process (W, £) = {W n ,(; n } n £N by observing 
the continuous time process just after the instants of service completions and 
the ends of switching times. Here W n € £l e identifies the parameters imme- 
diately after event n and £ n denotes the ordered pair of queue lengths so 
£ n S Nq (where No = {0, 1, 2, 3, . . .}) and specifically, £ n .i is the queue length 
at the server's current station and £ ni 2 the queue length at the other open 
station. 
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Let ei and e2 denote the unit vectors of M 2 . Given W n = (k,i,G,X), the 
possible increments of the queue lengths at event n are of the form 

sei + t&2i s, t € No during a switching time 

and 

sei + te2, sS {-l}UNo;f GNo during a service epoch 

(as the server's queue is always given by xi). During switching times and 
during service epochs when £ n = (x\,X2) with x\ > 1, the parameters will 
not change, that is, W n+ \ = W n but if, during a service epoch, we have 
x\ = 1, then an increment (— l,t) ends the current epoch. For the continuous 
time process, as stated earlier: station k closes; the server starts the switch 
to station i; some station j, selected using the Markov chain {I r }, opens; 
given the pair a service time distribution G and arrival rates A are 

selected using measure Vij\ the customers waiting at station i become the 
initial queue at the server's new location and during the switching time 
customers arrive at the open stations with the selected rates. The 

possible transitions for our embedded process are to states 

G, X;x2 + s,t) with intensity Pijdfij(G, \)hki(s,t; A), 

where hki(s,t;X) is the probability of increment sei + te2, given the arrival 
rates A, at the transition where the server switches from station k to i. 

In order to study the stability of the queueing system, we will consider the 
trajectories of a particle moving according to drift vector fields D related to 
the parameter regeneration process. As such vector field or fluid models are 
quite standard in the queueing literature, we will leave details for later and 
briefly describe now only what we need to state our main results. The one- 
step mean drifts of £, conditioned on the parameters in the service epochs, 
form a vector field D° : O x N — > M. 2 defined by 

(1) D V, r) = E(U + i ~ U | w r ) = (--!,-), 

where co r = (i,j, G, X±, A2). The process £ is homogeneous during each service 
epoch so the mean drift does not depend on queue lengths £ n but only upon 
oj t = (i r ,i r+ i,G r ,\ r ), the parameters in epoch r. Our vector field model 
will not include switching times so we only define the mean drifts for service 
epochs. 

Now we observe that if a particle starts from position (x, 0) and moves 
across a quarter plane in direction D°, then it eventually arrives at point 
(0,x\2/(n — Ai)). The multiplying ratio that appears here plays a key role 
in the phenomena we are interested in. For later use we observe that, given 
open stations 

(2) YlJ = ^- = tane, 

A* - Ai 
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where is the minor angle between D and the vector —ex- We also introduce 
the notation 

Fij(x) = Vij[(G, Ai, A 2 ) : Yij <x] for x > 

for the cumulative distribution function of the slope Yij for each pair of 
stations i, j G = {1, 2, . . . ,d} with i ^ j. For notational convenience, set 
Fu{x) = 0, x < 1 and Fa{x) = 1, x > 1, for all i £ 5^. 

Regularity conditions. In order to obtain our results, we have to impose 
some restriction on the regeneration of parameters and we state these now. 

Condition E. For all pairs of distinct stations i, j, there exists niQ > 0, 
Mq > mo such that: 

(i) z^[(G,A):Ai+mo<A£<M ] = 1, 

(ii) Fy(l/M ) = 0, F ij (M ) = l. 

Part (i) ensures that the service rate is uniformly bounded from above and 
below and that the server is never trapped at any station forever. Part (ii) 
limits the effect that any single epoch can have on the long term behavior 
of the process by preventing the mean drift vectors D(u;,r) from approach- 
ing the axial directions too closely (so it is a type of uniform ellipticity 
condition). 

By observing the vector field model only at parameter regeneration times 
we obtain a multiplicative Markov process of independent interest which we 
now define. 

Multiplicative model. Let (I,X) = {(I r ,X r ) :r > 0} denote a discrete 
time Markov process on the state space Sd x R + (recall that Sd = {1, 2, . . . , d}) 
which jumps from state (i,x) *— ► (j, ax) with transition measure PijdFij(a) 
for a > 0. The matrix P = [Py] is the transition matrix of an irreducible 
Markov chain on Sd and the Fij are a family of distribution functions with 
support 1R+ that satisfy Condition E(ii). Assuming that the process waits 
for time x in state (i,x) before making its next jump, we associate with 
(I,X) an accumulated time process {T r } defined by 

r 

(3) T = and T r = T r _ x + X r = J2 X n, r = l,2,.... 

l 

The sequence of times {T r } is increasing and so the limit T = lim^ooT,. 
exists (in some cases it is infinite) and we call it the total time of the mul- 
tiplicative model. 
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2. The main results. Let it = (711, 7T2, . . . , TTd) denote the equilibrium dis- 
tribution of the open stations Markov chain {I r }- For each pair i, j £ <S|, let 
lij be a random variable with distribution Py and for any constant s > 0, 
set 

(4) L^^PyEOogyy) and My (a) = PyE(l^)- 

J'=i 

The d x d matrix M(s) = [My(s)] is nonnegative and irreducible because 
{I r } is irreducible. Hence, M(s) has a Perron-Frobenius eigenvalue 77 (s) > 
such that 7](s) > \r)i(s)\ for all other eigenvalues of M(s). The size of rj(s) 
determines which moments of the total time T of the multiplicative model 
exist, which in turn affects the long term behavior of the random walk £. 
It is important in Theorem 3.1 below to note that 

d 

(5) r,'(0) = J2^Li- 

i=l 

We briefly indicate why this is the case. We have My(0) = PyE(logiy) 
and we note that the matrix M(0) has 77(0) = 1 with corresponding left 
eigenvector tt and right eigenvector 1. For any fixed s > 0, let u(s) be the 
left eigenvector corresponding to eigenvalue r](s) of M(s). Scale u(s) so that 
(tt — u(s)) -1 = (• indicates the scalar product). Now, for s small, write 
u(s) = 7r + sx, where x -1 = 0. We note that M y (s) = Py(l + sE(loglij)) + 
0(s 2 ) and r]{s)=u(s)M(s)l{u(s)iy 1 =u(s)M{s)l. 
Now we expand u(s)M(s)l to get 

rj{s) = (tt + sz)[Py (1 + sE(log^))]l + 0(s 2 ) = 1 + s £ + 0(s 2 ) 

since xPl = x • 1 = 0. The relation rj(s) = 1 + s J2 ^i^i + 0(s 2 ) immediately 
implies (5). 

The multiplicative Markov chain's time component and our calculations 
for it are key elements of our main results which give stability conditions for 
the queue length component £ = {£ n } of the stochastic model (W, £). Let 
denote the state (0,0) where both open queues have no customers and let 
r = min{n > : £ n = 0} be the time until the system first reaches this empty 
state. In both of the following theorems it is assumed that the measures 
for selecting the parameters at the starts of epochs satisfy Condition E. 

Theorem 2.1. (i) //Ei^i < 0, then P(r < 00) = 1. 

(ii) If '^iTjij = and switching times are all 0, then P(t < 00) = 1. 

(iii) If YA"K% L i > °> then p ( r = 00) > 0. 
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Remark. The case where Y^i^i^i = and switching times are nonzero 
is difficult and we do not consider it here. The two station model in this 
case but with constant, not regenerated, parameters was considered by Men- 
shikov and Zuyev in [11]. A detailed classification of a critical random walk 
on a 2-dimensional complex, again with a fixed jump distribution on each 
face, was given by MacPhee and Menshikov in [8]. In these papers, gener- 
alizations of the results of Lamperti on processes with asymptotically zero 
drift (see [6]) were used to establish how the behavior of the process during 
switching events affects which moments of hitting times to the empty state 
exist. 

The major result in this paper is that we can make refined statements 
about the tail of the hitting time r in the recurrent case (i) of the previous 
result by determining which moments of r are finite and which are infinite. 
The existence of moment s of r requires moment conditions on the service 
and switching times, specifically that there is some constant K s such that 
Jq°° X s dG(x) < K s uniformly over GeI and, similarly, x s dHij(x) < K s 
for all pairs of stations i, j. 

Theorem 2.2. IfYA^iU < 0, then: 

(i) ifn(s) < 1 and the sth moments of switching times and service times 
are uniformly bounded by some constant K s , then E(r s ) < oo; 

(ii) if rj(s) > 1, then E(r s ) = oo. 

This result implies that the region of parameter space where the queue- 
ing process £ exhibits "null-recurrence" can be of considerable volume. It 
also implies that we have only polynomial ergodicity in the sense that the 
time of reaching the empty state has only finite order moments and the order 
smoothly changes when we pass from the null recurrent to the positive recur- 
rent case. Note that as long as they have sufficient moments, the switching 
times play no role whatever in the result which is in accord with what is 
known about the constant parameter polling network in the recurrent case; 
see [5]. The phenomenon of a large null-recurrence region is present even in 
the case of zero switching times. We now illustrate it with a simple example 
before further commenting on it. 

Example. There are d = 2 nodes and the arrival rates are Ai = A2 = 
1 in all epochs. When the server is at station 2 service times are always 
exponential with rate 2. At station 1 service times are again exponential, 
but there are two possible service rates, 5/4 and 5. Each time the server 
comes to station 1 it works at the smaller rate with probability p £ [0, 1] and 
at the larger rate with probability 1 — p. That is, according to our notation, 

Ku,[(M,A):/i = 2,A = (l,l)] = l, 
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i*i[(/x,A):p = 5/4,A = (l,l)]=p, 

i*ji[(M,A):ai = 5,A = (1 > 1)] = 1-p. 

Now, elementary computations show that L\ = (4p — 2) log 2, L 2 = 0, it\ = 
7T2 = 1/2 so that Y^^i^i < is equivalent to p < 1/2. Further, Mu(s) = 
M 22 (s) = 0, Mi 2 (s) = l and M 2 i(s, )=pA s + (1 - p)(l/4) s so that 

? ? (s) = [p4 s + (l-p)(l/4) s ] 1/2 and r/(0) = (2p - 1) log 2. 

By Theorem 2.1, the process is certain to return to the empty state when 
J2^iLi < 0, that is, p < 1/2 and has positive probability of never returning 
when p > 1/2. The condition p < 1/2 implies that rj'(0) < and so rj(s) = 1 
has a root 

log(l-p) -logp 

s p = i — 5 > 

log 4 

in this case. By Theorem 2.2, E(r s ) is finite when < s < s p , and is infinite 
when s > s p . We see that s p = 1 for p = 1/5 and, thus, the system is transient 
for p€ (1/2,1], null recurrent for p£ [1/5,1/2) and positive recurrent for 
pG [0, 1/5), so, indeed, we have a "thick" null recurrent phase here. 



In order to establish stability results for the queueing system, we need 
various Lyapunov functions. We can find them by studying the long term 
behavior of particles moving according to some random vector field models 
built using the parameter regeneration process and, in particular, the mean 
drift vector field of the process £ over service epochs. 

The phenomenon of thickness of the critical region implied by the The- 
orem 2.2 and exhibited in the previous example is quite exceptional in the 
theory both of stochastic processes and of critical phenomena. This phe- 
nomenon appeared in a related model but with fixed parameters that was 
studied by Fayolle, Ignatyuk, Malyshev and Menshikov [4] (their results 
also appear in Section 5 of [3]). Their model is a random walk £ on a 2- 
dimensional complex, that is, a collection of quarter planes connected at 
their edges in some fashion with possible multiple connections and with no 
edge unconnected. On any quarter plane the walk drifts away from one axis 
and toward the other. When the boundary is reached, the quarter plane to 
enter next is chosen according to a transition matrix P. Specifically, The- 
orem 5.3.2 of [3] establishes recurrence of the Markov chain £ in a fixed 
environment (which satisfies our Condition E) if and only if the condition 
J2i KiLi < holds (in fact, their result is established also for cases where the 
transition matrix P is not irreducible). Further, Theorem 5.3.4 of [3] shows 
that under the same regularity conditions £ (again with fixed parameters) 
is positive recurrent if and only if 77(1) < 1. 
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The new features in this paper are the following: all the parameters in the 
FIMM model are known constants, while here they change at each switch- 
ing event according to some known distributions; moreover, the results in 
[4] concern only estimates of E(r), while here we have conditions for decid- 
ing whether E(r s ) < oo or not for any s > 0. Finally, the phenomenon of 
thickness of the critical region both in [4] and in the present paper is due 
to some underlying randomness in the choice of the next set of parameters. 
In [4] this randomness is due to a Markovian choice of the next complex, 
analogous to our matrix P = [Pij): when this randomness is eliminated, the 
critical region shrinks to a zero measure manifold. In the present paper the 
thickness persists even when the Markov process induced by the matrix 
P = [Pij] becomes a deterministic routing process. 

3. The multiplicative model and the vector field model. We first estab- 
lish our results for the multiplicative model. We then define the random 
vector field model and show that our results for the multiplicative model 
enable us to describe its behavior. Using the vector field model, we can de- 
fine the "random" Lyapunov function we use to establish our results for the 
stochastic process (W, £). 

Let (/, X) denote the multiplicative model and a a = min{r > : X r < ^4} 
for any given constant A > 0. Part of the next result appears in [2] where 
conditions were found determining when the expected total time of (I,X) 
is finite. 



E(T S 



Theorem 3.1. Suppose that the multiplicative Markov chain (I,X) has 
distributions that satisfy Condition E above: 

(i) IfJ2i^iLi < 0, then X r — > a.s. as r — > oo, the total time T of (I,X) 
is a.s. finite and for any s > satisfies 

< oo, ifn{s)<\, 
= oo, ifrj(s)>l. 

(ii) // Y^i^iLi = 0, then, for any A > 0, P(<ta < oo | Xq = xq) = 1 for 
any finite Xq. 

(hi) IfY^i^iLi > 0, then X r — > oo a.s. as r — > oo. 

Remark. It is important to note that n(0) = 1 and n is log-convex in s 
so r](s) < 1 for some s > is equivalent to r/(0) = J^^i^i < 0- If Si^i-^i > 0) 
then T = oo a.s. 



3.1. Proofs for the multiplicative model. We prove first the results about 
the limiting behavior of the process {X r } and then establish conditions for 
the existence of E(T S ). 
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Proof of Theorem 3.1 (Limiting properties of {X r }). The process 

(/,logX) = {(/ r ,logA r .):r>0} 

is a nonlattice random walk on Sd x R which is homogeneous in its second 
coordinate. Its jumps in this coordinate are bounded due to Condition E(ii). 
As we now show, its stability conditions can be demonstrated using the Lya- 
punov function f(i,x) = en + logs, where the en, i = 1, . . . ,d, are constants 
to be determined. For all states (i,x), let 

A = E(/(X r+1 ) - f(X r ) I I r = i, X r = x). 

Writing a = (a±, . . . , a^) , L = (L\ , . . . , L^) and (3 = , . . . , (3d) > we find that 
(3 = — (I — P)a + L, where I denotes the d x d identity matrix. The Markov 
chain {/ r } is irreducible with equilibrium vector tt so it follows that the 
matrix I — P has rank d — 1 and that it is orthogonal to the range of I — P. 
Hence, by standard linear algebra arguments, for any e > 0, there exists a 
such that: 

(i) (3i < —e for each i only when J2i=i ^i^i < 0; 

(ii) Pi = for each i only when Ylt=i = 0; 
(hi) i3i>e for each i only when J2i=i ^iLi > 0. 

Applying Theorem 2.1.7 of [3], we see that log X r — > ±oo a.s. or, equivalently, 
X r — s- oo or a.s. according as to whether J2i=i ^i^i is positive or negative. 
If X^f=i 7r i-^j = 0> then there exist constants ai such that {f(I r ,X r )} is a 
martingale with bounded jumps. By standard results, for example, Theo- 
rem 8.4.3 of Meyn and Tweedie [12], the process f(I,X) is recurrent and 
as 

{X r < A} D {f(I r ,X r ) < a min + log A}, 
the a.s. finiteness of a a follows. □ 

Remark. A similar model, a Markov chain on the half-strip Sd x Z+, 
was studied in Section 3.1 of [3] and we borrowed the Lyapunov function 
x + aj from there. 

We now consider the question of the total time of the multiplicative pro- 
cess. We start by restating its definition (3) in a slightly different way. For 
each pair (i,j) G 5| with i ^ j, Fij is a distribution function satisfying Con- 
dition E. We construct a sequence of random variables a±, ct2, ■ ■ ■ tied to the 
Markov chain {/ r } as follows: when / r _i =i, I r = j, then ct r has distribution 
function Fij. The Qj are conditionally independent given the state of {I r }- 
The total time of the multiplicative model started from state (i, 1) is 

oo r 

T = 1 + a\ + a\(X2 + aia2«3 H = 1 + X! II 

r=l 1 
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If Hi^i-Li > 0, then the set of states (i,x) with x > 1 is visited infinitely 
often and the total time T will be infinite a.s. When J2i ^iLi < 0, we can 
calculate which s > have E(T S ) < oo. 

Recall the d x d matrix M(s) = [PjjE(Y^-)]jj, where Yij has distribution 
function Fij. Let Vi(s) denote the right eigenvectors and rji(s) the corre- 
sponding eigenvalues of M(s). As E(Y^) > for all pairs i, j and {I r } is 
irreducible, the results of Perron-Frobenius state there is a simple eigenvalue 
n(s) > such that i](s) > \rji(s)\ for all other eigenvalues and its eigenvector 
v(s) has elements v^\s) > and is the only eigenvector with this property. 

We start by determining the asymptotic behavior of E(ai • • • a n ) s . 

Lemma 3.2. For any s > 0, there exist constants C > 0, 7 6 (0, 1) such 
that 



Proof. Let u be the row vector with U{ = P(/o = i) an d 1 denote the d- 
vector of Is. Following the standard argument for establishing the Chapman- 
Kolmogorov conditions, we can write 



and the result follows directly from this for each fixed s using the result of 
Frobenius. □ 

This estimate makes it evident that the value of rj(s) is crucial to deciding 
whether E(T S ) is finite or infinite. 

Proof of Theorem 3.1 [Conditions for E(T S ) = 00]. The proof that 
E(T S ) = 00 when n(s) > 1 is straightforward. As the on are a.s. positive, we 
have for any n 



If n(s) > 1, then E(cti • • • a n ) s — ► 00 as n — > 00 by Lemma 3.2 and so E(T S ) = 
00. When n(s) = 1 and s > 1, we have T s = (J2^=i a i ' ' ' a n) s > J2^=i( a i ' " ' a n) s 
and again the result follows from Lemma 3.2. 

When rj(s) = 1 and s < 1, we can use Jensen's inequality applied to f{x) = 
x s which says that, for any u, v, G, H > and s < 1 with G 7^ H, 



n 



lim n(s)~ n (E(ai ■ • • a n ) s ) = C + 0( 7 n ). 



V( ai ---a n y = u(M(s)) n l 



E(T s )>E( ai ---a n ) s . 




or 



(6) 



(uG + vH) s >{u + vY^iuG 3 + vH s ). 
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Let T n = 1 + a\ + a\ot2 + ■ ■ • + a\ ■ ■ • a n and let t n = E(T^). Applying (6) 
with u = 1, v = n~ a , G = T n and H = n a a\ ■ ■ ■ a n +i where a > 1 is constant, 
we obtain 

t n+1 = E(T n s +1 ) = E((T n + «!••■ a n+ i) s ) 

> (i + n^J'-^EC^) + n^" 1 )E(a 1 • • • a n+1 ) s ). 

From Lemma 3.2, there exist constants C > 0, 7 £ (0, 1) such that the term 
E(«i • • • a n+ i) s = C + 0(7") and so 

t n+1 > (1 + n"^" 1 ^ + Cn^" 1 )) + 0( 7 n ) 

> (1 + (s - l)n~ a ){t n + Cn^" 1 )) + 0( 7 n ) 

= t n + (s - l)n- a t n + CK^" 1 ) - (1 - s)n a ^) + 0( 7 "), 

where (l + re - " 1 ) 5 - 1 > l + (s-l)n" a as s < 1. Choose a = (l-s/2)/(l-s) > 1 
and for some constant /3 G (0, C/s), pick ng = uq(C, s) such that, for n > ng, 

(1 - s)(^n~ a+s / 2 + Cn a(s ~ 2) ) + 0( 7 n ) < (C - /?s/2)n- 1+s / 2 . 

This is possible as —a + s/2 < —1 + s/2 and a(s — 2) < a(s — 1) = — 1 + s/2. 

As t n >t\= E(af) > for all n, we can choose € (0, t no n s ^ 2 ). Hence, 
t n > I3n s / 2 for some n > uq and for such n, 

tn+i > Pn s/2 + Cn- l+s ' 2 - (1 - s)(/3n" a+s / 2 + Cn^" 2 )) + 0( 7 ") 

>/3 ^/2 + £ n -l+^ >^( n + l)-/2 j 

so, by induction, t n > /3n s / 2 for all n > uq. Finally, as T s > T£ for all n, 
E(T S ) = 00. 

Conditions for E(T S ) < 00. Now we assume n(s) < 1. If s < 1, then for, 
any n, 

(00 \ s 00 

n=l / n.=l 

and E(T S ) < 00 by comparison with the tail sum J2 n >n v( s ) n - 

It only remains to sort out the case when s > 1. Again, applying Jensen's 
inequality to the function f(x) = x s , we have, for any real numbers u, v, G, 
H > and s > 1 with G^H, 

(uG + vH\ s uG s +vH s 
\ u + v J u + v 

or 

(7) (uG + vH) s Kiu + vy-^uG 8 + vH s ). 
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Fix any (3 G ((^(s)) 1 / 5 , 1), and apply (7) with u = 1, v = /? n+1 , G = T n , 
H = /?~( n+1 )ai • • • a n+ \ to obtain the inequality 

E(T* +1 ) = E(T n + f3 n+l x /r (n+ V ' • • «„+i) s 

(8) 

< (l + ( g n+1 ) s_1 (E(r*) + p n+1 (3- s{n+1) v(ai ■ ■ • «n+i) s ). 

From the definition of n(s) and, choice of (3, it follows that 
lim P~ s( - n+1 ^B(ai ■ ■ ■ a n+1 Y = 0. 

n^oo 

Again, as t n >ti = E(af ) > for any n, it follows that there exists no such 
that, for all n > no, 

r s(n+1) E(a 1 ---a n+1 ) s <t„. 
Now (8) implies that, for all n > no, 

Wi < (1 + /3 n+1 ) s " 1 (*n + /3 n+1 t„) = (1 + /3 n+1 ) s t„. 
Thus, for any n, 

oo 

E(T n s )<t no J] (l + /3 m+1 r<oo. 

m=no 

That E(T S ) < oo when r/(s) < 1 and s > 1 follows from this bound and the 
monotone convergence theorem. □ 

3.2. Definition and properties of the random vector field model. We now 
define a random vector field model {(S(uj,t),V(uj,t)) :uj £ ^5,t > 0} using 
the parameter process W. It is motivated by the one-step mean drift vector 
field defined in (1), but we define a more general version so we can consider 
perturbations of this natural field. This type of model is simple and standard, 
but we describe it carefully here as we use it to define the random Lyapunov 
functions we need to establish the properties of the random walk process £. 

There are no switching times in the vector field model. Consider a particle 
that moves around on 5j x with velocity field D = {D(o;,r) G R_ x 
M + :uj G £ls-, r £ N}, where the D(w,r) = (— di(u r ), (fe^r)) satisfy conditions 
equivalent to Condition E which we now specify. 

Condition E'. As before, let Fij denote the distribution function of the 
slope Y(uj,r) = (^(ctv) / 'd\(uj r ) and assume that there exists some constants 
mo, Mo such that 

Uij[di > m , ||D|| < M ] = 1 and that ify(l/M ) = 0, Fij(M ) = 1 

for each pair i, j. 
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For our regenerated parameter process (W,£), the natural vector field 
satisfies d\ = 1 — Ai//i, d2 = X 2 /^ and we denote it by D°. In this section we 
are considering d\ and (L2 to be more general functions of the parameters. 

The process (S, V) on the state space £l e x N x has components 

S(u,t) = (i r ,i r+1 ,G r ,X r ,r), 

the parameter set at time t including the current epoch number and 

V(u,t) = (vi(u,t),v 2 (u,t)), 

the position of the particle on the quarter plane indicated by the parameter 
i r at time t. For each parameter sequence uj and initial point (xo,0) with 
xq > and to = 0, set S(uj, t) = S(u, 0) = lo± = (ii,i 2 , G±, Ai, 1) and 

V(u,t) = (x 0) 0) + tD(w,l) 

until the particle reaches the boundary of quarter plane (ii,^) which defines 
the end of the current epoch. This occurs at time 

ti = t 1 {u,x ) 



di (wi) ' 

Switching times are assumed to be in this model so at t = t\ set 

S(w,ti) = W2 and V(u,t 1 )=(d 2 (u 1 )-^^,0^={x Y{u;,l),0), 

where Y(uj,r) = d2(u} r )/dx(u r ). The components of V are transposed at t\ 
as the particle moves from plane (11,12) to plane (12,13) at that moment, 
corresponding to the server switching stations. For epochs r = 2, 3, . . . , 
define their end times by 

(n\ 4- -t- ( \ + j v l(u,U-l) 

(9) t r = t r {LU,Xo) = t r -i + 



di(u) r ) 

Now, for t G [t r -i,t r ), we set S(u,t) =u> r and 

V(w,<) = V(w,t r _i) + (t - t r _i)D(w,r) 

so that V2(oJ,t) = d 2 (oJ r )(t — t r _i) and then 

V(u,t r ) = (d 2 (u r )(t r - i r -l),0). 

It is clear from these definitions that, for every parameter sequence u, the 
sequence {t r } is increasing and so t (uj, xq) = lim^oo t r (u, Xq) exists. Under 
conditions such that t& is bounded a.s., it is the time taken for the particle 
to reach the empty state 0. We now determine conditions which ensure 
||V(o;,t)|| — * and conditions under which ||V(o;,£)|| — ► 00 as t — > 00 and 
then consider the process time. 
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The sequence vi(u, t r ), r = 1, 2, . . . , describes the distance of the particle 
from the origin at the starts of epochs r + 1. For each r = 1, 2, . . . , 

v 1 (uj,t r ) = d 2 (u r )(t r - t r -x) =d 2 (u r ) Vl ^fl tr ~^ =v 1 (uj,t r - 1 )Y(u,r) 

ai{io r ) 

and, hence, 

r 

(10) vi(u,t r ) = x Y[ Y(uj,n). 

n=l 

We re-iterate at this point that this vector field model is a random process 
driven by the parameter regeneration process including the Markov chain 
{I r } for the server location. Recall that when I n = i, I n +i = j 7^ i, then 
Y{<jj,n) = d 2 (u n ) I di(u n )) ~ Yy with distribution Fiji ir = (tti,tt 2 , ■ ■ ■ ,^d) de- 
notes the equilibrium distribution of {I r }', Li = £)f=i i^jE(loglij) for each 
i. For C>0, let t c = min{i > : vt(u, t) + v 2 (u, t) < C}. 

Lemma 3.3. For the random vector field model (S,V): 

(i) Sj=i TTi-C/j < 0, then v\{uj,t r ) —> Q a.s. and 

t (uj,xo) = lim t r (a;,xo) < oo a.s., 

r— >oo 

(ii) if E?=i ^ = 0, toen P(i c < oo) = 1, 

(iii) £/ X)j=i fi-^i > 0j then vi(uj , t r ) ^ oo a.s. asr^oo. 

Proof. The process {(/ r , t>i(u;, i r ))} r is a multiplicative Markov pro- 
cess, so parts (i) and (iii) follow immediately from Theorem 3.1(i) and (iii). 
Theorem 3.1(h) implies that min{r : Vi(u,t r ) < C} is finite u-&.s. From equa- 
tion (9) epoch r lasts for time t r — t r —\ =vi(u,t r —i)/di(u r ) which is finite 
as di(u> r ) > mo and, hence, tc < oo a.s. □ 

Remark. As t r (u, xq) = xot r (tv, 1) for all r and all xq > 0, it follows that 
if t (uj,l) < oo, then t (uj,XQ) < oo for all xq > 0. 

Next we recall the Perron-Frobenius eigenvalue rj(s) of the matrix M(s) 
defined in (4). 

Lemma 3.4. Suppose Ya=x ^i^i < 0. For s > 0, ifr](s) < I, then B(t s ) < 
oo while if rj(s) > 1, then E(t^) = oo. 

Proof. Combining equations (9) and (10), we obtain 



16 



MACPHEE, MENSHIKOV, PETRITIS AND POPOV 



for r = 1, 2, ... and so 

oo oo 2 r 

tsa = Y\U = 2; oE T7 T II Y (u,n). 

r=l r=l zv r ' n=l 

Under Condition E' for the vector field model, there exists a constant K > 
(uniformly in u) such that 1/K < l/c^uv) < X a.s. for all epochs r. It 
follows that 

oo r oo r 

j? e n y ( w > n ) < ( w . ^o) < tfzo e n y ( w > n )- 

r=l 1 r=l 1 

The expression Erllti^( w ! n ) is the total time for the multiplicative ran- 
dom walk associated with the parameter regeneration process that drives 
this vector field model. From Theorem 3.1 (i) , it follows that E(i^) < oo is 
equivalent to rj(s) < 1. □ 

The assumptions in Condition E' ensure this construction can be carried 
out for indefinitely many epochs for all parameter sequences from any initial 
particle position x with x > in any epoch r. To indicate this, we denote the 
switching instants defined above by t n (u), r, x) for n>r. Under the condition 
J2f=i KiLi < 0, it follows that t (uj, r, x) = lim n ^ 00 t n (u>, r, x) < oo and is the 
time taken by the particle to reach from initial point x in epoch r. In this 
case, for any constant A > 0, 

(11) t A (u,r,x) = min{t:V(t) eB A ,S{0) =u r ,V(0) =x} < oo a.s., 
where = {x € : x + y < A}. 

4. Lyapunov functions and their properties. Now we return to studying 
(W, £), the random walk together with its regenerated parameter process. 
We use the vector field model to construct Lyapunov functions along each 
parameter sequence co and then take expectations over the parameter process 
to establish Theorems 2.1 and 2.2. We outline the simplest case of our scheme 
now to help explain this general argument. Let E w denote expectation for the 
random walk conditioned on the parameter sequence oj, let (r n ,£ n ) denote 
the epoch and queue lengths at time n and let t a = min{n : £ n 6 B^}. For the 
natural vector field D°(w,r) = (— d\(u, r), d^iuj, r)) = (\ r ,i/fJ, r — ^iK^/Hr) 
consider the function t A defined in (11). For oj such that t A is finite and for 
x^B A , 

(12) E u (^(w,r,^ +1 ) - t A (u,r,£ n ) \ £ n = x) = -1. 

In the case where switching happens instantaneously, Foster's criterion (see 
Theorems 2.1.1 and 2.2.3 of [3]) implies that 

Eo,(ta |£ = x ) <*A(^)l)Xo) <oo 
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and it follows, by taking expectation over and using Lemma 3.4 with 
s = 1, that E(ta I £0 — x o) < 00 whenever r/(l) < 1. 

We see from this that, during epoch r, ^(w, r, •) based on D° is a random 
(w-dependent) Lyapunov function for establishing stability (E(ta) < 00) of 
the queue length process ^. Our plan is to show that when E(i^) < 00 
(conditions for this are given in Lemma 3.4), then E(-r|) < 00. However, a 
stronger result than Foster's is necessary to decide whether E(-r|) < 00 for 
given s > 1 (see Theorem 5.3 below). This result requires stronger bounds, 
than (12). To get these stronger bounds we actually have to use tA from 
a vector field which is a modification of D°. It is also necessary to handle 
nonzero switching times. 

To establish which moments of ta are infinite, a different approach is 
needed. We show that for almost all parameter sequences uj there is some 
probability, (3 > 0, say, that the random walk takes time comparable with 
the vector field to reach B^- This (3 is uniform over uj, so we can take 
expectation over to show that E(i^) = 00 implies E(r^) = 00. 

4.1. Modified vector fields and Lyapunov functions. Consider the set of 
particle trajectories, under a vector field D, from all possible initial points 
on the ei axis in epoch 1. For any parameter sequence uj, any point (r, x) € 
N x Ml lies on a unique such trajectory. 

Definition 4.1. For a vector field D satisfying Condition E', we define 
two distinct Lyapunov functions. For each parameter sequence uj, define 

g:fi s xNxl^l + by <?(w,r,x)= 7 , 

where 7 is such that the trajectory under ~D(oj,-), started from (1, 7ei), 
passes through the point (r, x), that is, g is constant along particle trajec- 
tories under the vector field D. On 

F = {wen s :i (w,l,ei) < 00}, 

the set of parameter sequences for which t is everywhere finite, define 
/ : F x N x R% -f R + by 

f(uj, r, x) = t^(w, r, x) where A > is a given constant. 

We will need to consider expectations E w of / and g as functions of the 
random walk £ for given uj and, subsequently, with respect to the measure 
on the parameter sequences. 

Remark. For any given uj, the functions / and g are linear within 
epochs and we consider their properties for general vector fields D. We 
will call / the remaining life, as it is the time for the particle to reach 
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= {x € R+ : x + y < A} from initial point x in epoch r along the tra- 
jectory defined by the vector field D for the parameter sequence co £ F. 
Similarly, we will call g the initial value function and observe that it is 
everywhere finite for a.s. all to for any D satisfying Condition E'. 

In particular, we will use / for establishing that specified moments of the 
random time ta = min{n : £ n € B^} exist, g for establishing instability and 
/ and g together for showing nonexistence of certain moments of ta- We 
now describe some of their properties. 

Lemma 4.2. If vector field D has Yji^iLi < 0, then Pn(F) = 1. For 
fixed u), the functions f (for u> E F) and g are linear in x for fixed r and 
continuous at changes of epoch r. Writing: 

(i) f(u, r, x) = a r x\ + b r X2 + fo for uj £ F (a r , b r and fo are functions 
of u), we have: 

• a r = b r _i for each r and there exists M > 0, uniform in to and r, such 
that b r /M < a r < Mb r , 

• (j) r =max{a r ,fe r } > 1/Mq (the constant in Condition ~E'J, 

• f(u,r,x) > (xi +x 2 )/M ; 

(ii) g(uj, r, x) = c r x\ + c r+ \X2, we have: 

• c r+ i = c r /Y(uj,r) for each r and, hence, c r /c r+ \ =Y(uj,r) £ (1/Mq,Mq) 
for all epochs r, 

• c r Iln=l n ) = c i f or a M r - 

PROOF. Pq(F) = 1 is the content of Lemma 3.3(i). That /, for uEF, 
and g are linear in x for fixed r and continuous at changes of epoch r follows 
from the vector field D being constant during epochs. 

The properties of / for cj & F follow from Condition E' which constrains 
d\ and d 2 and the relations a r {x\ — d\) + b r {x 2 + d 2 ) = a r x\ + b r x 2 — 1 
or, equivalently, 1 + b r d 2 (r) = a r d±(r) for each r. The lower bound on / is 
immediate. The relations satisfied by the c r follow similarly. □ 

Unfortunately, a r and b r , the x-coefficients of / within an epoch, can 
be arbitrarily large, so the level curves of f(tv,r,-) (constructed under D°) 
in the x-plane can make an arbitrarily small angle with the natural drift 
direction D°(w,r) of {£ n }- This prevents us making the estimates needed to 
establish which hitting time moments exist if we construct / with respect 
to field D°. 

Modified vector fields. To resolve this problem, we construct modified 
vector fields (small perturbations of the natural vector field D°) for which 
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/ and g permit some useful estimations. We do this in different ways for 
showing whether E(-r|) < oo or not. 

To show that E(r^) < oo, we want a vector field D such that its remaining 
life / is greater than that of D° but is still finite a.s. and for which moment 
s of / exists. When we want to show that E(r^) = oo for some s, we use a 
different modification of D° and the initial value function g. 

Definition 4.3. As in equation (2), we write 

9 (uj,r) =arctan[A 2 (w,r)/(^(cj,r) - \i(u,r))] 
for the minor angle between D°(w,r) and — ei. We define 
D + (w,r) = (-4(w r ),4(w r )) 

(13) 

= \\n (oj,r)\\(-cos(9 (uJ,r) + 9'), S m(6 (uj,r) + 6')) 

for some 9' > such that 9o(uj,r) + 9' < ir/2 for all pairs uj, r. We will say 
that such a D + is uniformly above D°. We further define 

D~(u;,r) = (-d 1 (uj r ),d 2 (uj r )) 

(14) 

= ||D V r) || (- C os(9 (uj, r) - 0'),sm(0 o (uj, r) - 6')) 

for some sufficiently small 6' > 0. In this case we say that D~ is uniformly 
below D°. 

This definition is possible, as, by Condition E, 

9o(u,r) € (arctan mo/Mo, arctan Mo/mo) C (0, 7r/2). 

Clearly, there are uj for which the remaining life under D° is finite, but 
under D + it becomes infinite. In fact, the set of these has zero probability for 
some suitably small 9' as we now show. Let M + (s), M°(s) and M~(s) denote 
the matrix [i-^-E(Y^)] under the vector fields D + , D° and D~ respectively 
and let f? + (s), rj°(s) and tj~(s) denote the Perron-Frobenius eigenvalues of 
these matrices. Also, for each station i, let L® = J2j=i -fijE(logl^j) under 
D° and Lr the same quantities under D . 

Lemma 4.4. For any given s > 0, we can choose 9' > small enough 
that: 

(i) if rf(s) < 1, then r/ + (s) < I; 

(ii) if rj°(s) > 1, then T)~(s) > 1; 

(iii) if J2i nL°i > 0, then J2i inLi > 

(9' does not depend upon uj). 
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Remark. When J2i n i^i < an d 0' is chosen to satisfy condition (i) or 
(ii), we say that D + or D~ respectively is s -neutral. 

Proof of Lemma 4.4. For the vector field D + , we have Yy = d£ jd\ = 
tan(#o + &') when uj r = (i,j,G,X). As the function tan is continuous on 
(0,7r/2) and n + (s) is a continuous function of M + (s), it follows that ij + (s) 
is continuous in 9'. Part (i) follows as Condition E bounds the rate of change 
of n + (s) in 9' and parts (ii) and (iii) follow from a similar argument. □ 

The advantage of working with D + rather than D° is that the expected 
one-step jumps of the process {/(w, r n , £ n )}n are of size comparable to 4> r 
where, as before, f(ui, r, x) = a r x + b r y + fo and 4> r = max{a r , b r }. Let Af n = 
f(u,r n+ i,£ n+ i) - f(u,r n ,£ n ) and Ag n = g(u, r n+u £ n+1 ) -g{u,r n ,£ n ). 

Lemma 4.5. Suppose that D + lies uniformly above D° and is s-neutral 
for some s > 0. Let f denote the remaining life of D + . Then F + = {uj € 
O5 : f(uj, l,ei) < 00} satisfies Pn(F + ) = 1 and for each to € F + and any 
service epoch r, 

E(/(w,r,x) s ) <oo 

and there exists £\ > such that 

E LU (Af n |r n = r,£ n = x) = -a r d\(u),r) + b r a%{oj,r) < -£\<j> T , 

where 4> r > 1/Mq. Similarly, the initial value function g of a vector field D~ 
which is uniformly below D° satisfies 

E tJ (Ag n I r n = r,£ n = x) = -c r d^(uj,r) + c r+ i^(u;,r) > e 2 c r 

for some £2 > which does not depend upon r and there exists a constant 
k > such that 

E LU (Agl I r n = r,£n = x) < c ^k. 

Proof. P(F + ) = 1 and E(/(w, r,x) s ) < 00 follow immediately from 
Lemma 3.4 applied to the vector field D + . 

Let (3, (3 + denote the angles between the vector (a r ,b r ) and D , D + re- 
spectively. By comparison with (12), we see that > ir/2 and, by construc- 
tion, there exists 9' > such that f3 + = (5 + 9'. The inequality for E(A/ n ) 
follows from the comparability of a r and b r , 4> r > 1/Mq (from Lemma 4.2), 
||D°|| > mo and the cosine rule. 

The inequality for E(Ag n ) follows from a very similar argument, while the 
existence of the constant k for bounding E(A#„) follows from the restriction 
on the rates A imposed by Condition E and the assumption that service time 
distributions G are selected from a set V with uniformly bounded second 
moments. □ 
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5. Proofs for the polling system. We are now in a position to establish 
Theorems 2.1 and 2.2. Our treatment of the vector field model in the previous 
sections ignored switching times, but we must now consider them. 

5.1. Technique for handling switching times. Our main method is to ob- 
serve the process not at all transitions, but at carefully calculated sampling 
times. Theorem 5.1 below states the semi-martingale result from Fayolle, 
Menshikov and Malyshev [3] which we will use to establish the recurrence 
part of Theorem 2.1. 

Let {Zt,t > 0} be a sequence of nonnegative random variables with Zq 
constant and Zt measurable with respect to a filtration J- = {J~t,t > 0}. Let 
{N n ,n > 0} be a strictly increasing sequence of stopping times adapted to T 
such that iVo = and |-/V n _|_i — N n \ is bounded for all n > 0. Define a sampled 
process {X n ,n > 0} by Xq = Zq and X n = Zjy n for n > 1. Also, for constant 
C > 0, let ( c = min{t > 1 : Z t < C} and a = min{n > 1 : X n < C}. Finally, 
let {X t /\ a } denote the sequence {X t } stopped at a. With this notation, we 
now state Theorem 2.1.2 of [3]. 

Theorem 5.1. If Z > C and for some e > and all n > 
E (^(n+i)A<r I FN nAcr ) < X nAcr - eE(A r ( n+1)A(T - N nAa | TN nAa ) a.s., 
then E(Cc) <Z /e <oo. 

To use this result, we will calculate in the proof of Theorem 2.1 a con- 
stant N such that the inequality of this theorem is satisfied N steps after 
any switching time. 

5.2. A key supermartingale for the instability results. We now show that 
the level curves of the initial value function g of D~ form a useful barrier 
for the random walk £. For each u S $7, K > 0, let 

(15) QkW) = {(r, xjeNxl^: g(u, r, x) < K} 

denote the set of states below the trajectory of D~ with initial value K. Let 

Pk{w) = min{n: (r n ,£ n ) G Gk{u)} 

denote the hitting time of the random walk to Qk ( r n denotes the epoch in 
which step n of £ occurs). We recall that = {x € : x + y < A}. 

Lemma 5.2. Let g denote the initial value function of vector field D~ 
and let ta denote the hitting time o/B^ by £. Let Z n = g(uj, r n ,£ n ) for n £ N 
and suppose that Zq = zq for some constant zq > A. Define U n = l/Z nATA 
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for n = 1, 2, If A is sufficiently large, then, for every epoch r and every 

parameter sequence uj, 

E w (C/ n+ i -U n |r n = r,£ n =x) <0 

and, hence, Uoo exists a.s. 

Proof. Taylor's theorem applied to the function (1 + provides the 
inequality 

(1 + x)' 1 - 1 < -x + x 2 (l - t )~ 3 at any x > -t > -1. 

Write Z n = z, r n = r and consider £ n = x ^ B^. From Lemma 4.2(h), we 
have z > c r A/M$ and either Ag n = Z n+ \ — Z n > or < —Ag n < c r . In 
either case Ag n /z > —Mq/A > — 1 for A > Mq. It now follows from Lemma 
4.5 and the assumption c r /z < Mq/A that 

E w (C/ n+ i - C4 | £ n = x) = - f E w f — — \ — - £n = x]-l 



z\ \l + Ag n /z 



<IeJ-^ + (i-MoM)- 3 ^# 

z V z z 

<-^ + d-M 0M )-3^. 

The right-hand side can be made negative by choosing A sufficiently large. 
Therefore, along any given parameter sequence, the process (U n ) is a bounded 
super-martingale which guarantees the existence of U<x>- d 

5.3. Proof of Theorem 2.1. (i) We wish to show that r, the return time 
to the empty state 0, is a.s. finite under the condition J2i ^i^i < 0. Consider 
a vector field D + which is uniformly above D° and s-neutral for some s > 0. 
As before, we denote the remaining life of D + by / and let F denote the set 
of parameter sequences where / is a.s. finite for all points (r,x). For each 
uj € F, Lemma 4.5 implies 

E w (/(u;,r,£ re+ i) - f(u,r,£ n ) \ = x) < -e\4> T < -ei/M 

during service epochs. During switching times, the queue lengths have drift 
upward so this inequality does not hold. Let T r _i denote the time at which 
the event which concludes epoch r — 1 occurs. Then 

E w(/(w,r,Cn+i) - f(u,r,£n) I £n =X,T r _i = 71, UJ r = (i,j,G,X)) 
= (a r Xi + 6 r A 2 )mjj > 0, 

where to™ is the mean switching time from station i to j. To employ The- 
orem 5.1, we choose e\ > according to Lemma 4.5, e' G (0,£i) and an 
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integer N > 2Mom*/(ei — e'), where m* = m&Xijmij. As long as x = (x,y) 
has x > N, 

E w (/(w,r,£ n+ jv) - f(u,r,£ n ) \ £ n = x,T r _i = n,u; r = (i,j,G,X)) 

(16) 

< 0r[(Ai + A 2 )m* - £l iV] < -e'cp r N < ~N 

Mo 

as 4> r = max(a r , b r ) > 1/Mq and Ai + A2 < 2Mq by Condition E. 

Now we apply Theorem 5.1 to Z n = /(w,r n ,£ n ), with C chosen so that 
ta, the time hits B^, satisfies ta < (c (the time {Z n } falls to level C). 
This is possible because, by Lemma 4.2, f(u>,r, x) > ||x||/Mo, that is, \\x\\ 
large implies / large. Hence, 

/(^,0,x)Mq 
E^t^ I Co = x) < < 00 

for each w£f and this in turn implies that the process £ enters B^ infinitely 
often along almost all parameter sequences. Our process is not irreducible in 
the normal sense, as it is not possible to reach the empty state in epoch 
r from state (r, x), where x = (x,y) with y > 0. However, it is possible to 
reach in epoch r + 1 and the probability of this is bounded away from 
uniformly in uj (the bound does depend upon A and Mo). Hence, by Borel- 
Cantelli, we have r < 00 a.s. 

(ii) Here ^7TjLj = 0. Again, we use / to denote the remaining time for 
the vector field model, but this time with the natural field D . We have 
from Lemma 4.2 that / is finite a.s. and from the inequality (12), we know 
that for each parameter sequence u € F the process (f(uj,r n ,£ n )) n is a su- 
permartingale (remember that we only consider instantaneous switching in 
this case). Now, from standard results, it follows that ta is a.s. finite for 
every uj G F. That r is a.s. finite follows as in part (i). 

(iii) We have ^ 7r «-^« > an d we wish to show that r = 00 with pos- 
itive probability. Consider the initial value function g for a vector field 
D~ uniformly below D° but with J^i^i^ > 0. We know from Lemma 
3.3 (iii) and Lemma 4.2(h) that, for almost all parameter sequences u, the 
x-coefncients c r of g in epoch r converge to as r — > 00. For such u and for 
any given constants A > and K' > 0, there exists ro such that [recall (15)] 
the set (r, B^) C Gk'(w) for all epochs r > ro. For any fixed parameter se- 
quence u>, the process £ is certain to reach some state (r',x') ^ Qk>{oj) where 
r' > ro- Let K = g(ui,r',x'). Starting from (r',x'), px> < ta a.s. by construc- 
tion but, as we now show, px> < 00 a.s. is impossible. The conditions of 
Lemma 5.2 hold from which we get E^^oo) < 1/K, but if px> < 00 a.s., 
then Uoo > 1/K' >1/K a.s. As 17^ > 0, it follows that P^ta = Pk> = 00) > 
1 - K'/K > for almost all u. 
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Remark. Another way to describe the idea in part (iii) of the proof 
above is to say when (r, B^) C Gk' f° r an epochs r there is some prob- 
ability (3 > (uniform over parameter sequences to) such that P w (£n ^ 
Qk 1 for all n) > /3, that is, if the vector field D~ has trajectories that re- 
main outside Qk> , then the random walk £ can stay outside Qk> indefinitely. 

5.4. Semimartingale estimates for Theorem 2.2(i). We now state the key 
result we use for establishing finiteness of moments of hitting times to the 
neighborhood of the empty state. It is a slight extension of Theorem 1 of [1]. 
This, in turn, is the generalization to general powers s of Theorem 2.2 of [7] 
(a similar result for integer moments only). 

Theorem 5.3. Let K> 1, (O, (J r n ) n ,P) be a filtered probability space 
and (Z n ) ng N an {J- n ) n - adapted positive valued process with Zq = z > K. 
Let ok = inf{n > 1 : Z n < K}, a' an arbitrary (J- n ) n - stopping time and a = 
inf{<7^-, a'}. Suppose that there exists sq > such that, for every n E N, we 
have E(Z*°) <oo. If there exists £2 > such that 

E(^ +1 - I T n ) < -£2^T 1 on {a > n}, 
then, for all s < so, there exists C = C(sq, £2) such that ~E(a s \ Zq = z) < Cz s . 

Remark. If we try to apply this result to Z n = f(u},r n ,£ n ), where / is 
the remaining life of the natural field D°, the condition of the theorem is 
not generally satisfied. This is why we have to introduce the modified field 
D+. 

Proof of Theorem 5.3. Straightforward extension (to deal with the 
introduction of a') of the corresponding proof of Theorem 1 of [1] . □ 

The following result establishes the key inequality for the application of 
Theorem 5.3 under a condition on r)°(s), the Perron-Frobenius eigenvalue 
of M°(s) which was considered in Lemma 4.4. As in the proof of Theorem 
2.1(i), it is necessary to sample the process at carefully chosen times to deal 
with the absence of service during switching times. 

Proposition 5.4. Suppose that rj°(so) < 1 for some so > and let D + 
be uniformly above D° and SQ-neutral. Write Z n = f(u, r n , £ n ) for all u £ F, 
where f denotes the remaining life under D + . Then for such to and all s < sq, 
there exists £2 > such that at times n which are not switching times and 
for ||x|| sufficiently large, 

E W (Z* +1 - Z s n I r n = r, £ n = x) < -£ 2 Z s n ~ l . 
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Proof. Letting z = Z n = f(u,r,x), we have, for ||x|| (and, hence, z) 
large, by Lemma 4.2, 1/C||x|| < (f> r /z < C/||x|| for some C > which is 
uniform in u. Writing Af n = f(u},r n+ i,£ n+ i) - /(u,r„,£„), we have 

E W (Z^ +1 - Z s n | r n = r, £ n = x) = E w ((« + A/„) s - z s | r n = r, £ n = x) 

and from Taylor's theorem, 

{z + Af n ) s = z s + sAfnZ 3 - 1 + s(s - 1) /" (A/ n ) 2 (z + tA/ n ) s ~ 2 (l - t) eft. 

JO 

As A / n > —(j) r , when r n = r, we have, for t e [0, 1], 

/ ^, \ s-2 

^ „s-2 | 



(z + tAf n ) s ~ 2 < z s ~ l ( 1 - — ) if s < 2 

and so, from Lemma 4.5, 

E(Z* +1 - | r n = r, £ n = x) < 4> T ( - S£l + C lS (s - 1 



where C\ > is some constant which is uniform in uj. 
For s > 2, we have 

(z + tAf Y~ 2 < i zS ~ 2 ^ + *) S " 2 ' if A/ " ^ z ' 

( + «W < { (A/ n )- 2 (l + t)- a , if A/ n > z. 

The assumption of a uniform bound K s on moment s of the service time dis- 
tribution implies there is some constant C such that E W (A/* | r n = r, £ n = 
x) < and, hence, E w (A/^/{A/ n > z} \ r n = r,£ n = x) = 0(<#). In 

this case, for ||x|| large enough, there exists some C\ > (uniform in cj) 
such that 



E(^ +1 -Z«|r n = r,en = x)<^ s -^- Sei + C lS ( S -l)^ + 0((</) r /z) 

again using Lemma 4.5. 

Finally, for any given s > 2 and any £2 with < £2 < sei, we can choose 
||x|| large enough that — se± + C\s(s — + 0(((/) r ./z) s ~ 1 ) < —£2. The ar- 
gument is similar for s < 2. This establishes the result as 4> r > 1/Mo- □ 

Proof of Theorem 2.2(i). As in Proposition 5.4, we consider the 
process f(uj,r n ,£ n ) where / is the remaining life under a vector field D + 
which is uniformly above D° and s-neutral. It follows from E(/(w, r, x) s °) < 
00 (Lemma 4.5) and the fact that E w ((£ n i + £ n 2) s ) < 00 for all lo, s > and 
all n that E(/(cj, r n , £, n ) s ) < 00 for all s 6 [0, so] and all n. 

To deal with switching times, we sample this process at times N n where 
iV n+ i — N n = 1 unless N n is the end of an epoch, in which case N n+ i — 
N n = N for some constant N to be determined. This makes it necessary to 
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introduce modified versions of the hitting times for which we wish to make 
moment estimates. 

Let Z n = f(u, r Nn ,£ Nn ), a K = inf{?i > 1 : Z n < K}, f A = inf{n > 1 : £ Nn 6 
Ra} for suitably large A and a = min{(T^-, f^}. For Zq = z > K for some 
sufficiently large constant K, we have, by Proposition 5.4, 

(17) VUK+i - K \r n = r, £„, = x) < -e 2 Z'- 1 on {a > n} 

as long as jump N n is not a switching event. If the server switches queues 
at time N n , then 

Z n +i = Z n + f(uj, r Nn + 1, Cn„+n - €N n )- 

By the argument leading to the inequality (16), we have, for large enough 
N (but £jv n > N to ensure there is no change of epoch), 

E(f(u,r Nn + l,tN n+N - CnJ I F Nn ) oc -N. 

Using this together with the moment bound assumed above, it follows that, 
for given £2, there exists ./V such that the bound (17) also holds when the 
jump at time N n is a switching event. 

Hence, by Theorem 5.3, there exists a constant C such that E w (<r s | Zq = 
z) <Cz s uniformly over co 6 F. We can choose A > KMq which ensures that 
Tj4 < ok a-s. and, hence, for x ^ B^, 

E(fl|£o = x)=E(a s |Z = z) 

= E(E w (a s |Z = z)) 
<CE(/(w,l,x) s ) <oo. 

To complete the proof, let ta = ini{n > l:£ n G B^} for suitably large A 
(so ta is measured in process time n rather than the sampling sequence 
{N n }) and recall that r = inf{n > 1 :£ n = 0}. Observe that ta < Nta so 
that E(-r| I £0 = x) < 00. Finally, by a standard argument, for example, 
Theorem A.l from [10], it follows that E(r s | £0 = x) < 00 for all finite x. 
□ 

5.5. Proof of Theorem 2.2(h), nonexistence of moments. We start this 
proof by showing that, with a probability that can be bounded away from 
uniformly for all parameter sequences, the random walk started outside Gk 
stays outside Gk at least until it hits B^. 

Proposition 5.5. Suppose that r)°(s ) < 1 for some s > and that g 
is the initial value function of an s-neutral vector field D~. For any given 
parameter sequence ui, suppose that the queue length process £ starts at state 
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(r, x) such that g(u,r,x) = K and x ^ for A large enough to satisfy the 
conditions of Lemma 5.2. For 1 < K' < K , 

PUpk'>ta)>1-^>0- 

Proof. As r)°(so) < 1, Theorem 2.1(iii) implies that ta is a.s. finite. 
Consider the process U n = l/g(£nAT A Ap K ,)- By Lemma 5.2, (U n ) is a positive 
super-martingale. Therefore, the limit Uoo exists almost surely and its law 
is supported by [0, 1] as K' > 1. Moreover, by Fatou's lemma, 

^ = E„(C/ )>E Ci; (C/ oo ) 

= jt]~Pu J (pk> < t a ) + Ew(U TA I p K ' > ta)~PUpk> > ta) 

as ta is a.s. finite. Hence, P u (pk' < t a) < 7^ < 1 by the initial choice of K 
and K' and (U n ) being nonnegative. Thus, P u (pk' >t~a)>1 — > 0. □ 

Proof of Theorem 2.2(h). We have n°(s) > 1 and r/°(so) < 1 fo r some 
so < s - Hence, the remaining life / of vector field D~ (uniformly below 
D°) is a.s. finite. As / and the initial value function g of D - are both 
linear within epochs, g can be scaled so that g(uj, l,xei) = f(u, l,xei) for 
all x > 0. Starting ^ from state (r, x) with g(uj,r,x) = K > K' and applying 
Proposition 5.5, we see that Puj(pk' — t a) > 1 — K'/K > uniformly over 
all uj. 

We now show that on the event {pk 1 ^ t a} the random walk takes such a 
long time to reach that the sq moment of ta cannot be finite. The event 
{pK 1 ^ t a} can only happen when (i) there is an epoch f where (r^B^) is 
not a subset of Gk 1 -, that is, where the D~ trajectory {g(u, S(t), V(t)) :t > 
0, g(ui, S(t),V(t)) = K'} enters B^ during epoch f; (ii) the random walk 
does not enter Qk 1 prior to epoch r. 

For fixed uj and K' , suppose the random walk starts from £o = (xo,0) in 
epoch 1 with g(tj, l,xoei) > K' . Let the random times T±, T2, ■ ■ ■ denote the 
ends of the epochs (for the random walk, not the dynamical system) up to 
epoch r — 1. On the event {px> > ta} we have 

r-l 

/(w,l,x ei) <Y^tr - t r -i + AM /m , 

r=l 

where, in the notation of Section 3.2, the t r are the epoch endtimes for a 
particle moving according to the vector field D™ and the bound AMq/vtiq 
on the final epoch's contribution to f(u, l,XQei), the time for the particle 
trajectory to reach B^, is obtained from Condition E with some obvious 



28 



MACPHEE, MENSHIKOV, PETRITIS AND POPOV 



geometry. Recalling the notation vi(u,t r -i) for the particle position at the 
start of epoch r (for the dynamical system) and writing X r _i for the queue 
length at the start of epoch r (for the random walk), we have, for each epoch 

r < f, 

T r — T r _i > X T —\ > v\(lo, £ r _i), 

as only one job can complete in each service time and, by assumption, the 
random walk does not enter Gk>- From 

vi(u, t r -i) = d\(t r — t r -i) > rno(t r — t r _i), 

we have 

f— l f— l f— l 

r A > T?_i = ^T r -T r _i > ^2vx(u,t r -i) >m ^2t r - t r -i 

r=l r=l r=l 

> mof(u, 1, x ex) - AM 

on all random walk trajectories such that ta < Pk' ■ 

Finally, as i]~(s) > 1, for any initial vector x ^ for A sufficiently large, 

E(r| | Co = x) > (1 - A7A>igE([/( W , 1, X' ei ) - AM ] S ) = oo, 

where the final equality follows from Lemma 3.4 applied to the vector 
field D-. □ 
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